Outlook: nix/OS etcd controller 'case-study'/exploration (Part 0)
Preface
This post is more a loose collection of thoughts / ideas I am currently exploring very much on the side w.r.t. my ongoing other project of kubernetes with / on nixOS. Its idea was mainly born out of the cumbersome ergonomics of deploying/join HA control-plane nodes (nixOS) where the new member CP node runs apiserver(N)+etcd(N) where nixOS rebuild switch on remote hosts does not make itself "eventually consistent" because a runtime reconfiguration of etcd requires some basic reconciliation.
The exploration is controller runtime agnostic which means it does not have to as kubernetes controller inside a kubernetes. This choice is initial to keep it simpler to iterate and worry less about controller/kubernetes boilerplate and also the context which motivated this exploration assume no "base" kubernetes cluster. These are only initial assumptions/conditions and I am very open and plan to explore a later integration into a model where this runs in a bootstrap kind cluster and / or might be used as a component for a Cluster API provider.
Loose ideas
Layer A: Controller purely based on etcd grpc communication with a relatively static desired state via spec from nickel. Layer B: Controller purely based on nix (flake, nixos-configuration) to build minimalistic nixOS with etcd Layber C: Controller/EventBus/TO-BE-DECIDED purely to orchestrate communication of Layers A and B.
Layer A and Layer B need then to be connected to trigger appropriate reconcile
Side-Note: try etcd as unikernel
The Journey so far
Most of this including the etcd unikernel has been built as proof of concept and revealed certain shortcomings already, that are mostly related to how I implemented it code-wise. Currently there are two control(ler) planes, i.e., the etcd gRPC management and below a libvirtd etcd microkernel machine provisioner. The machine provisioner would need to be implemented based on what a machine constitutes, e.g., for the initial nixOS VM scenario above no controller exists yet. The management controller and the "spec" which represents a cluser+nodes has been extended to include the concept of a Machine, which can be External/Remote or just a Machine. The former assumes externally managed etcd members and only on/offboarding is done by the controller and the magic of starting a learner member only after it has been registered is offloaded. For a Machine the underlying machine controller takes care of spawning a new VM via libvirt in the same domain/subnet as the existing members are in for demo purposes.
Caveats
Reconciliation: Reconciliation designed and implemented in its current state desires to be improved. There is a designed End of Reconciliation, which real world controllers do not have in the same sense, but for demo/development purposes the controllers and their runtime are designed to terminate once "done". This has to change. Spec changes are not support during execution. This is also a big missing feature. Member<>Machine controller communication is very bare-bones and in-process via tokio channels. Most of this can be addressed by moving this into the realm of Kubernetes and re-using primitives from their and make it Kubernetes operators. Since the point of this is however, to make bootstrapping and extending kubernetes control planes on/with nixOS easier this has not been done yet. Reasons already stated: bootstrapping issue. We can partially circumvent this by having the operators run in a disposable kind cluster that we spawn ad-hoc though.